Systems and methods for photogrammetric rendering

ABSTRACT

Systems and methods are provided for producing a rendered drawing or rendering from a detailed image of an object (e.g. photograph) resulting in a rendering that is photogrammetric and that preserves detail in the said image of said object. The combination of the metric nature and image detail preservation in a rendering resulting from the process enhances the usefulness of the rendering to users. The invention is useful in particular for large format renderings such as wire frame style drawings used for blueprints in the architecture, engineering and construction industry (AEC industry) when used for existing structures. The processes combine graphic arts techniques with photogrammetric techniques to preserve, fully or partially, information about an object as captured in image detail of said object and to present said information in photogrammetrically correct rendering, which rendering may be incorporated into drawings useful to and/or familiar to end users of said drawings.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/800,159, filed on May 4, 2007, which claims the benefit of U.S.Provisional Patent Application No. 60/797,511, filed on May 4, 2006,both of which are hereby incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION

Traditional 3D computer models, color laser printing, andphotogrammetric based maps and images have provided photographic andother forms of representations of buildings, terrain, objects, andartifacts.

Traditional drawings used in the architecture, engineering andconstruction industry (AEC industry) take the form of blueprints.Traditional blueprints are one-color print documents of an object.Traditional blueprints provide one-color line drawings that use a wireframe representation of an object and are also sometimes referred to as“line art” or “spot color” in the graphics art industry. Suchtraditional drawings may be adequate in the AEC industry for newconstructions as the object of the drawing may not yet exist (i.e., maynot yet be constructed). Such traditional blueprints and line artdrawings, however, are deficient as such traditional drawings onlyprovide minimal information in the wire frame about the object. It istherefore desirable to provide line drawings with enhanced detail.

Traditional photogrammetric techniques for the AEC industry producerectified and orthogonally corrected images (“orthophotos” or“orthoimages”), partially rectified, or unrectified images that may beconverted into two Dimensional (2D) wire frame models primarily throughmanual processes on a drafting board or partially manual processes withthe aid of a computer assisted drafting board or computer tableau (e.g.,CAD/CAM).

Additionally, traditional photogrammetric techniques can be used withCAD/CAM to produce three dimensional (3D) models in either wire frame ortextured surface models. Such models can be used for buildings and otherobjects. End-user blueprints in some cases cannot be created from thesetechniques, however. As a result, these techniques are deficient. Forexample, 3D models are difficult to depict in a visually useful way on a2D surface (e.g., paper) and textured surface models are in acontinuous, black and white tone that does not reproduce well byblueprint, fax, and photo copy commonly used in the AEC industry.

Traditional models and images also tend to be relatively large in filesize and consume large amounts of storage space and transmissionbandwidth. 2D wire frame models, while easy to reproduce as blueprints,have eliminated much useful information from original photographic orother images. Accordingly, blueprints that provide only line art in onecolor are widely used due to the deficiencies of such 2D and 3D modelsbut are yet still deficient in their own right.

As discussed above, traditional drawings used in the AEC industry ofexisting structures are wire frame style line art drawings produced asblueprints. After a structure is built, drawings are traditionallyredrafted to reflect the structure in its “as-built” condition. Largeformat photocopiers are used in the AEC industry to produce “recorddrawings” of newly completed structures when hand written notations on“working drawings” of completed projects must be preserved in multiplecopies for architects, owners, and contractors.

Redrafting projects using CAD/CAM to as-built conditions, however, islabor intensive and costly. Detailed features (e.g., stone masonryjoints or facade surface detail) is not included in “as-built” drawingsbecause of the difficulty of creating accurate images by traditionalCAD/CAM methods, difficulty or impossibility of displaying informationin blueprint form, costs, and other factors.

FIG. 1 shows prior art manual process 100 that manually takesinformation from field measurements of a physical object (e.g., step110) and metric photography (e.g., step 112). Information from boththese sources is then merged in a manual process (e.g., step 114) toprepare a CAD/CAM drawings production (e.g., step 116). During/after thepreparation of some/all of a CAD/CAM drawing of an object, measurementson the drawing are manually correlated with the field measurements takenof the object (e.g., step 118). Manually gathering field measurementsand manually matching field measurements to the drawing scale is laborintensive. The manual process produces a wire frame style drawing (e.g.,in step 199) that is a 2D drawing. Such a 2D drawing is reproduced in aone-color blueprint representation of the object. Such a process isdeficient as each of the several steps of gathering field measurements,merging photographic information with physical measurements,cross-checking scale drawings to field measurements, and the actualproduction of the CAD/CAM drawing itself is labor intensive. It istherefore desirable to provide a less labor intensive process offabricating drawings.

A prior art processed image is shown in image 1100 of FIG. 11, which isabsent of detail of stone texture 1012 of FIG. 10 (loss of detail 1112)and paint drops 1014 of FIG. 10 (loss of detail 1114). Also it isdifficult to determine if area 1118 on FIG. 11 is masonry or wood. Oneskilled in the art would likely determine that area 1112 in FIG. 11 isthe wood sill of the window and based on its proximity and the formfactor of the abutment to 1118, that 1118 is probably wood trim exteriorframing. Such a non-definite determination, however, is disadvantageous.

SUMMARY OF THE INVENTION

Systems and methods are provided for producing a rendered drawing, orrendering, of an object that is reproducible in one color. The rendereddrawing, or rendering, may be generated from a detailed image of anobject (e.g. a color photograph). The rendered drawing, or rendering,may be photogrammetric and reproducible in one color (e.g. blueprint,monochrome laser print), yet still preserve a large amount of the detailwith respect to the amount of detail in image (e.g., a photograph).

The metric nature and amount of image-detail preservation provided insuch a rendering may significantly enhance the functionality of therendering to users. Accordingly, such renderings may be utilized in verylarge-scale formats, such as wire frame style drawings used in the AECindustry for existing structures.

Presenting an object in greater detail in a drawing provides a user ofthe drawing with an enhanced interpretation of pre-existing conditions,with respect to said object, in “as-built” or “as is” environments. Suchan increase in detail allow for an improved determination, by a user (orcomputer), of particular locations on the object when compared totraditional less-detailed wire frame style drawing. Such improved detailmay find particular benefit with respect to the industry of repairingmasonry facades as it is beneficial to show greater detail in order toenhance a mason's understanding of the structure of an object that needsrepair or the structure of an object that was recently repaired.

Particular graphic arts techniques are realized along withphotogrammetric techniques to preserve, fully or partially, informationabout an object that was captured in image such as a photograph. Suchtechniques allow for the information to be presented in aphotogrammetrically correct rendering. Such a rendering may be utilizedto fabricate a particular type of drawing—such as a drawing usefuland/or familiar to mason or AEC user.

A photogrammetric rendering is provided that includes a digital imagerectified photogrammetrically and rendered using particular graphic artstechniques. Such a photogrammetric rendering may be utilized tofabricate a particular type of drawing and may be combined with othertypes of information (e.g., additional images, legends, type or otherindicia) to form the particular type of drawing (e.g. a blueprint).

In producing photogrammetric renderings by combining particular graphicart techniques and photogrammetry to produce rectified images ororthoimages incorporating “as-built” object detail many advantages maybe realized. For example, nonmetric photographs and images may beutilized to prepare photogrammetric renderings, such as historicaldocuments for objects that may no longer exist. As per another example,particular types of drawings, such as blueprints, may be economicallyproduced. As yet another example, detailed digital renderings may bequickly rendered, stored, and printed. As yet another example, detail ofexisting objects not previously expressed in a rendering may now beexpressed and incorporated into a rendering. Non-contact photographicmeans may also be utilized in fabricating detailed renderings. As peryet another example, the quantity of verification by field measurementneeded may be reduced or eliminated. As per yet another example,economical production of end-use drawings containing photogrammetricrenderings may be realized as compared to CAD/CAM preparation from photoor field measurements.

A detailed photogrammetric rendering may be produced to scale orout-of-scale. Additional information such as written notes may be addedto the renderings.

Particular graphic arts techniques may be utilized with photogrammetrytechniques to produce drawings incorporating photogrammetric renderingsderived from the processing of non-metric images. Such drawings may beproduced either as actual blueprints or in a format directly and/orclosely analogous to blueprint paper drawings familiar in the AECindustry (e.g. laser print).

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the present invention willbe apparent upon consideration of the following detailed description,taken in conjunction with accompanying drawings, in which like referencecharacters refer to like parts.

FIG. 1 shows a prior art flow chart for a manual method of producing adrawing resulting in a wire-frame model.

FIG. 2A shows a flow chart for producing a photogrammetric renderingpreserving features and detail visible in a photograph.

FIG. 2B shows a flow chart for producing a photogrammetric renderingfrom an image pre-processed by particular graphic arts techniques topreserve features and detail visible in a photograph.

FIG. 3 shows a flow chart for producing a photogrammetric renderingpreserving features and detail visible in multiple photographs withintermediate storage of rectified digital photogrammetric images priorto joining into a single digital photogrammetric image and prior tographic arts rendering.

FIG. 4 shows a method for producing photogrammetric rendering from aphotograph using multiple graphic arts processes to correct aphotogrammetric image before rendering and possible further graphic artsprocessing.

FIG. 5 shows an raw image from a non-metric camera of a building andbackground to be processed.

FIG. 6 shows a processed image in which extraneous objects andbackground have been removed through graphic arts processing (e.g.,physical masking and negative duplication of unmasked area or inelectronic photo editing highlighting and then deleting areas ofdifferent contrast).

FIG. 7 shows a rectified image correcting for camera lens distortion andimage distortions due to perspective, skew, etc (e.g., using graphicarts cameras where the original image plane can be rotated from parallelto the image receptor plane to take a photo that removes perspective orin digital photo editing stretching the photo along vertical orhorizontal axes at one edge).

FIG. 8A shows a rendering that has been processed according to thepreserving feature detail from photographs and produced in such a waythat the line art image could be used as a substitute for blueprints(e.g., in digital photo editing finding edges of contrast change andeliminating other image content).

FIG. 8B shows an image that has been processed according to inventionpreserving feature detail from photographs and produced in such a waythat the grayscale image could be used as a substitute for blueprints(e.g., eliminating color information through the uses of color filtersor in digital photo editing converting to grayscale).

FIG. 9A shows a close up view of the rendering in FIG. 8A showing moreclearly the detail in existing objects.

FIG. 9B shows a close up view of the image in FIG. 8B showing moreclearly the detail in existing objects.

FIG. 9C shows a close up view of a portion of the image in FIG. 7rendered to enhance deep shadow contrast features (e.g., produced withflash under exposure on photo duplication).

FIG. 9D shows a close up view of a portion of the image in FIG. 10rendered to enhance shadow contrast features.

FIG. 9E shows a close up view of a portion of the image in FIG. 7rendered to enhance shadow low midtone features.

FIG. 9F shows a close up view of a portion of the image in FIG. 7rendered to enhance shadow high midtone features.

FIG. 9G shows a close up view of a portion of the image in FIG. 7rendered to enhance shadow highlight features (e.g., produced with flashover exposure).

FIG. 9H shows the combination of FIGS. 9C through 9G.

FIG. 10 shows a close up of a portion of the image in FIG. 7.

FIG. 11 shows a prior art wire frame drawing of the object in FIG. 10.The wire frame drawing is typical of CAD/CAM in the level of detailinformation provided about the object to a viewer of the drawing.

FIG. 12A shows a grayscale rendering of FIG. 10 after processing toeliminate color information and convert to continuous tone grayscale andthen reduce the number of levels of gray used.

FIG. 12B shows a rendering in a technical drawing style similar in formto wire frame drawings produced by CAD/CAM, but preserving a greaterlevel of detail than shown in FIG. 11, yet still be reproduced as lineart (e.g., a blueprint).

FIG. 12C shows a rendering in which areas of the rendering are filled toenhance the visibility of certain features of the object (e.g., producedin digital photo editing using one or more of finding edges,posterization, specifying pixel count or curve radii in lines or linethickness).

FIG. 12D shows a pen and ink style rendering which provides greaterdetail than FIG. 11, yet can still be reproduced as line art (e.g., ablueprint).

FIG. 13A shows a grayscale image of FIG. 10 after processing with atechnique to find and enhance edges of contrast or texture change inFIG. 10, and then further processed to alter image density levels inhighlights, midtones and shadow areas (e.g., reduce saturation ofshadow, highlight contrast changes, increase saturation of shadow inhighlight areas, reduce midtone saturation while holding shadow inmidtones).

FIG. 13B shows further processing of FIG. 13A to reduce the image fromcontinuous tone to a small limited number to color levels and then tofurther reduce the prior result to 1 bit color (e.g., reducing graylevels to 4 and then converting all gray levels to solid black). Thisfigure is highly suitable for reproduction as one color line art (e.g.,blueprints).

FIG. 13C shows FIG. 10 with further reduction of highlight image densitylevels as comparable to FIG. 13A prior to reducing the number to levelsto two, the same as that in FIG. 13B.

FIG. 13D shows FIG. 13C further processed to reduce the image tomonochromatic 1 bit color by converting all gray levels to solid black.

FIG. 13E shows processing similar to FIG. 13C but with a more limitednumber to levels of image density (e.g., shadow detail is almost nonexistent while minimal highlight detail is preserved with the mainfeature being changes in contrast or edges). Note the reduced level ofavailable detail as compared to FIG. 13C.

FIG. 13F shows processing similar to FIG. 13C, but with reduction to twolevels of image density resulting in a monochromatic 1 bit color depthsuitable for reproduction as line art (e.g., blueprint).

FIG. 14A shows a rectilinear projection of a gird which is analogous,for example, to an object in a photograph when taken with a metriccamera or other image produced by other means, or when effects show inFIGS. 14B through 14G have been corrected. Such a figure may be utilizedas, for example, an elevation drawing by an architect.

FIG. 14B shows sample distortion due to light refraction distortion dueto the camera lens in photography of a type know as barrel lensdistortion.

FIG. 14C shows an effect essentially opposite of FIG. 14B due the samereasons, but called pin cushion effect.

FIG. 14D shows rotation of an image on a film or digital image due tothe camera and object not being horizontally aligned on the same visualhorizon. FIG. 14E shows skew effect in photography along the horizontalaxis when the right side of the object being photographed is furthersway from the film or image receptor plane than the left side.

FIG. 14F shows skew effect in the vertical axis when the top is furtherthan the bottom from the image receptor plane. This effect is usuallycalled perspective effect to distinguish it from skew as described inFIG. 14E.

FIG. 14G shows a multiplicity of effects.

FIG. 15 shows a flow chart for producing a photogrammetric renderingpreserving features and detail visible in multiple tonal areas of theoriginal image by processing tonal ranges such as shadow, midtone, andhighlight independently for graphics art processes with intermediatestorage of graphic arts processed images for a multiplicity of tonalranges prior to joining intermediate images into a single digitalphotogrammetric image.

DETAILED DESCRIPTION OF THE INVENTION

A photogrammetric rendering may be provided, for example, by processinga non-metric digital image (e.g. digital color photograph) to be fullyor partially rectified according to a photogrammetric technique. Suchprocessing may create an orthoimage that results in a digitalphotogrammetric image. Such a digital photogrammetric image may then befurther processed by a graphic arts rendering process that results in aphotogrammetric rendering.

All such processes may be performed by a computer autonomously. Once onesub-process completes, the next process may be automatically initiatedand performed by the computer. Alternatively, for example, a process maybe performed by a computer and the computer may request manualinstructions at each processing step or at only particular processingsteps. For example, a computer may request confirmation that aparticularly processed image is satisfactory and that the nextprocessing step should be initiated. The computer may allow a user toreject a particular image and enter in, or modify, parameters for theprocessing step that produced the image such that a more desirable imagemay be obtained. A user may instruct the computer to perform anyprocessing step at any time. A user may setup processing packages thatinclude multiple steps such that the user may initiate the package sothe computer can autonomously perform all of the steps in the package.The user may add, remove, or modify steps from a package. Such packagesmay be initiated over the Internet such that a customer may upload animage/images for a desired object/objects, select a particular package,and then have the opportunity to download the results of those packages.Different packages may have different prices associated with thepackages. Accordingly, a customer may upload an image that the userdesires to have a blueprint for, start the process over internet (e.g.,by pressing a START button) and then be presented with a blueprint thatthe customer can download over the internet. Accordingly, an autonomousprocess may be performed on a remote computer that can be fed animage/images over the Internet (or an intranet) and deliver processedimages over the internet (or an intranet). A customer/user may beprovided with the ability to add, remove, modify process steps to apackage. Licenses may be sold that allow a user to use a package,multiple packages, or all packages for a period of time (e.g., a year)or for a set number of images (e.g., 5000 or less). Some licenses may,for example, allow a user/customer to add/modify/remove process stepsfrom packages. The library of process steps available to a user/customermay, for example, change depending on the license.

Graphic arts rendering processes may include a variety of graphic artsprocessing techniques. Any number of graphic arts processing techniques(e.g., one, two, three, four, or more) may be performed in a graphicarts rendering process. For example, a graphic arts rendering techniquemay find contrast or texture transition zones (e.g., finding edges),reduce color images to grayscale, adjust the tonal balance (e.g.highlight, midtone and shadow balance adjustment), reduce continuoustones to a limited number of tones or levels (e.g. four-levelgrayscale), and equalize all tone levels to one level. Such techniquesmay similarly occur in any order or a particular order (e.g., the orderdescribed above).

A digital photogrammetric image resulting from a graphic art renderingprocess is a photogrammetric rendering. The photogrammetric renderingmay be provided, for example, in one color, such as a line drawing orwire frame style drawing. For example, the photogrammetric rendering maybe provided as a blueprint tailored to the AEC industry. Accordingly, aphotogrammetric rendering may be, for example, one color and capable ofreproduction as a blueprint.

Persons skilled in the art will appreciate that a multiplicity ofgraphic arts processes, in addition to or without the processesdescribed above, may be performed in order to provide a finalphotogrammetric rendering suitable to a particular user. Persons skilledin the art will appreciate that the graphic arts processing techniquesmay be ordered in any way (e.g., as described above) and reordered for aparticular image. Similarly, the order of the photogrammetric processand graphic arts processes may be performed in any order (e.g., graphicsart processes before photogrammetric processes and vice versa).Particular processes may be performed in the middle of other processes.For example, graphic arts processes may be performed betweenphotogrammetric processes (and vice versa). Additionally, particulargraphic arts processes may be applied to the original, non-metricdigital photograph prior to, and after, any photogrammetric correction.

FIG. 2A shows process 200 that may include non-metric digital image 222.Persons skilled in the art will appreciate that a metric digital imagemay be used in lieu of non-metric digital image 222. A metric image maybe captured by, for example, metric photographic equipment. Similarly,stereo photos or stereo images may be utilized in process 200 (e.g., inlieu of non-metric digital image 222) as such photos and images mayachieve better results in a photogrammetric correction process.

An initial image used to start process 200 may, for example, include anynumber of objects and any portion of any objects. Additionally, imagesmay be removed (e.g., cut) from images such that images of particularobjects may be separated and processed individually. Similarly,component parts of interest may be separated from an object of interest.All such objects and component parts may be reassembled after processing(e.g., after photogrammetric and graphic arts processing).

Additionally, an initial image may be digital or analog. For example,the starting image could be of a variety of types such as an originalhand drawing, a photograph that has been printed or otherwise reproducedas a halftone image, an intermediate image such as photographiccontinuous tone negative, a photographic continuous tone separation, ahalftone separation film, a printing plate, a painting, an originalartwork, or any type of image.

Regardless of the form of the starting image, an image may be, forexample, converted to a digital image the image may be stored as a fileon a computer. Persons skilled in the art will appreciate that any rawunprocessed, preprocessed, beginning, intermediate or final image,rendering or file may be stored temporarily or permanently on a varietyof media for further archival or further processing. Such images may bestored as a .JPG, .PICT, .TIFF, .BMP, or any other type of image file.Digital images may be, for example, be captured by way of a camera orscanner.

Non-metric digital image 222 may be processed through photogrammetriccorrection step 225 to provide digital photogrammetric image 230.Graphics art rendering step 235 may process digital photogrammetricimage 230 to provide photogrammetric rendering 245. Process 200 maycomplete at, for example, step 249, albeit additional processing stepsmay occur anywhere in process 200 (e.g., after photogrammetric rendering245 is obtained).

FIG. 2B shows process 250. A non-metric digital image may also be, forexample, pre-processed or post-processed with respect to aphotogrammetric correction process (e.g., step 225) to remove objectsother than the object, or objects, of interest. Objects not of interestmay be removed or eliminated from a base images by one or more of avariety of techniques that may include, for example, erasure, masking,threshold cutoff, tonal balance adjustment, or any other graphic artstechniques.

Process 250 may include non-metric digital image 252 that may undergopre-process graphic arts step 256 to provide pre-processed non-metricdigital image 258. Photogrammetric correction may then occur inphotogrammetric correction step 259 to provide digital photogrammetricimage 260. Graphics art rendering may process photogrammetric image 260to form photogrammetric rendering 285. Process 250 may complete at, forexample, step 299, albeit additional processing steps may occur anywherein process 250 (e.g., after photogrammetric rendering 285 is obtained).

Persons skilled in the art will appreciate that if reproduction of thephotogrammetric rendering requires printing as single spot color of ink,monochromatic 1 bit color, processes 200 and 250 may be, for example,suitable. However, if grayscale reproduction would reproduce greaterdetail, for example, the graphic arts rendering process (e.g., step 235or 265) may stop short of including indexing, equalization, or otherprocessing to a two-level scale (e.g., in digital images, one bitcolor).

The image being processing may, for example, be metrically scaled to aknown scale at any point after a digital photogrammetric image isachieved (e.g., digital photogrammetric images 240 and 260). Furtheringthis example, a scaled photogrammetric rendering may be processed into adocument of a format suitable for end-use (e.g., a blueprint), which maycontain additional information (e.g., a description of the object, itslocation, the date, compass orientation, a project name, drawing number,version number of the drawing). A final formatted document may thenprinted or otherwise pre-produced for end-use. Final printing may be inthe form of, for example, a blueprint if the image is 1 bit color depth.Grayscale photocopy or printing, for example, may be utilized if theimage is greater than 1 bit color depth.

Those familiar with the graphic arts will also note that many additionalprocesses to enhance contrast, feature definition, color balance, hue,color saturation and other aspects of the image may be accomplishedprior or during the preferred embodiment.

Additionally, for example, the use of colors may be utilized. Colors maybe referred to as, for example, color channels in digital imageprocessing, RGB (Red, Green, and Blue) used in video displays ofmulti-color images, or CYMK (Cyan, Magenta, Yellow, and Black) in printproduction. Such schemes may be utilized in processing images to achievecolorized results. For example, color channels may be processedmathematically whether in positive, negative, or inverted form andwhether singlely or in combination through additive, subtractive,multiplicative, or differential formula to improve and/or enhance finalimage results.

Images may be pre-processed in a variety of ways. For example, imageownership processing such as adding copyright information, securityconcerning image content, or segmenting processing may occur in order toretain intermediate images, increase productivity, impact economics, oraffect security. Such pre-processing may include, for example, theproduction of orthoimages from single or stereo images, joining or“stitching” multiple images together into a single image, or convertingimages from color to black and white (and vise-versa).

FIG. 3 shows process 300 that may use, for example, multiple startingimages. Persons skilled in the art will appreciate that it is oftenbeneficial to start a process of creating a photogrammetric renderingwith multiple images due to the size of the object of interest, thefield of view of the imaging device, any obstructions obscuring the viewof the object, convenience, or in order to achieve properphotogrammetric correction of projections from an object (e.g.balconies, gables, etc.).

Each image may, for example, be processed individually in order tocreate separate digital photogrammetric image. Such digitalphotogrammetric images may be stored temporarily or permanently as theyare processed. The multiple digital photogrammetric images may then bejoined together (“stitched”) to form a single image of the object ofinterest. The resulting single image may be photogrammetric in characterand may be further processed by graphic arts rendering in order to yieldthe desired photogrammetric rendering.

Persons skilled in the art will appreciate that additional graphic artstechniques of cutting, pasting, overlapping, erasing, and others may benecessary to result in an image of the whole object of interest found inthe multiple images.

In process 300, non-metric digital images may be obtained in step 340.An image may be chosen from the remaining to-be-processed images in step342. For the chosen image, photogrammetric correction may occur in step325 to form digital photogrammetric image 330, which may be stored instep 332. Step 344 may then determine if there is any image that isto-be-processed that has not yet been processed. If so, another imagemay be chosen and processed. Step 346 may determine if more than onedigital photogrammetric image was obtained and, if so, the digitalphotogrammetric images may be joined into a single digitalphotogrammetric image in step 348. Graphics art rendering may occur instep 335 to provide photogrammetric rendering 385. Process 300 maycomplete, for example, in step 399.

FIG. 4 shows process 400 where iterative multiple graphic arts processesmay be utilized after photogrammetric correction in order to achievedesired results in the post-processed photogrammetric rendering. Process400 may, for example, include one or more graphic arts processes thatmay be utilized after graphic arts rendering, which may result in apost-processed photogrammetric rendering or “PP PGR” 465.

A post-process graphic arts process on a photogrammetric rendering maybe, for example, the adjustment of tonal balance (e.g. highlight,midtone, shadow) in order to accentuate the edges of contrast change orto eliminate unwanted detail to allow a post-processed photogrammetricrendering to be reproduced as blueprint in one bit color.

Process 400 may include non-metric digital image 422 that may beprocessed by photogrammetric rendering 425 and graphic arts rendering435, the result of which may be valued at step 436 in order to determineif the result is satisfactory. If satisfactory, photogrammetricrendering 485 may continue to determination stage 460 to determine ifadditional processes 462 should occur to produce PP PGR 465. Process 400may end at, for example, step 499. If a satisfactory result is notobtained, one or more of the processes (e.g., all the processes) may beundone (reversed) in step 438. For example, the graphics art renderingmay be undone. Alternatively, saved versions of the file after differentprocessing steps may be retrieved and utilized (and the image desiringreversal may be deleted or ignored). Other graphics art processes mayoccur in step 453 and another determination as to whether the result issatisfactory may occur at step 454. If the result is satisfactoryphotogrammetric rendering 485 may continue to step 460. Ifunsatisfactory, the process can be redone at step 456 (or the apreviously processed image may be retrieved). Additional graphics artprocess determinations (e.g., step 458) and related steps may occur atany time.

FIG. 5 shows a non-metric digital image that would be typical of a rawunprocessed image that might be available to be used as an image tostart process 250 of FIG. 2B. Process 250 may, for example, start aphoto, image, or set of stereo photos, or stereo images of the object orobjects on interest to be shown in the final image or document. Image500 of FIG. 5 may, for example, be processed by removing or eliminatingextraneous objects such as backgrounds, foregrounds, objects other thanto one or ones of interest from a non metric digital image by one ormore of several techniques such as erasure, masking, or othertechniques.

The result of a graphic arts pre-process of removing objects not ofinterest of image 500 may be image 600 of FIG. 6. Persons skilled in theart will appreciate that image 600 has not undergone photogrammetriccorrection processing. In this case, for example, the graphic artsprocess of removing objects not of interest was accomplished in apreprocessing step prior to photogrammetric correction.

A pre-processed non-metric digital image is then processed to produce aphotogrammetricly corrected, rectified or orthogonally correct image, an“orthophoto” or “orthoimage”. Image 700 of FIG. 7 shows a digitalphotogrammetric image resulting from the photogrammetric correctionprocess on a preprocessed image, in which backgrounds and other objectsnot of interest have been removed to form the preprocessed image.

FIG. 8A shows the results of the graphic arts rendering process (image825) on image 700 of FIG. 7. FIG. 8B shows image 875, which is similarto image 825 of FIG. 8A only to the step of continuous tone grayscale.

Image 910 of FIG. 9A shows detail of image 825 of FIG. 8A also as arendering and image 920 of FIG. 9B shows detail of image 875 of FIG. 8Bin grayscale. The images of FIGS. 8A, 8B, 9A and 9B show the retaineddetail found in the original non-metric digital images as shown in FIG.5, 6, or 7. In the case of the AEC industry, this retained informationwould be quite useful and valuable to planning work such a masonryrestoration of the building pictured in the FIGS. 8A, 8B, 9A, and 9B.

FIGS. 9C through 9H show that it may be useful to render an image inmore than one tonal range to extract the maximum useful information fromthe original non metric digital image.

Image 930 of FIG. 9C, for example, shows rendering at a thresholdcut-off level for levels for deep shadow.

Image 940 of FIG. 9D, for example, shows rendering at a thresholdcut-off level for levels for shadow.

Image 950 of FIG. 9E, for example, shows rendering at a thresholdcut-off level for levels for low midtone.

Image 960 of FIG. 9F, for example, shows rendering at a thresholdcut-off level for levels for high midtone.

Image 970 of FIG. 9G, for example, shows rendering at a thresholdcut-off level for levels for highlight.

Thus, FIGS. 9C through 9G show rendering at threshold cut-off levels fordeep shadow, shadow, low midtone, high midtone, and highlightrespectively for 9C, 9D, 9E, 9F, and 9G.

FIG. 9H shows the combination of FIGS. 9C through 9G into a singlerendering. A process for creating the individual rendering for thevarious tonal ranges may be provided by, for example, process 1500 ofFIG. 15.

Process 1500 includes a non-metric image or digital image may bepre-processed to create digital photogrammetric image 1505. The tonalbalance of the image in shadow, midtone, and highlight ranges may, forexample, be determined and isolated in steps 1510, 1520, 1530, 1540,1550, and 1560, respectively. Persons skilled in the art will appreciatethat any number of threshold determinations and trace renderingprocesses may occur. For example, five thresholds may be determinedcorresponding to, for example, deep shadow, shadow, midtone, highmidtone, and highlight. Persons skilled in the art will appreciate thattrace rendering processes based on tonal thresholds may, for example,isolate separate sub-images (e.g., channels/layers of interest).

A graphic arts process, in this case, a rendering is performedseparately on all or some of the tonal ranges. Such a rendering mayoccur, for example, by finding particular edges (e.g., as occurs insteps 1520, 1540, and 1560). Thresholds may be set for such processesand may merely be turned ON or OFF by the user. Alternatively, forexample, a user may set any threshold. Each of the steps 1510, 1530, and1550 as well as 1520, 1540, and 1560 may vary for each tonal range toextract the most beneficial result for the users.

After the trace rending process, the resulting rendered trace images arethen merged back into a single image 1590. In digital image photoediting this process may be accomplished using layers. A result of sucha process may be, for example, merged Photogrammetric Rendering 1599.

The results of a process similar to process 1500 can be seen in images9C through 9H as follows. Here, five tonal ranges have been used andseparated into separate images. In each case, the rendering process usedis finding edges or highlighting the areas of contrast change or shift.The result of the process of merging the separate tonal ranges can beseen in image 980 of FIG. 9H.

Persons skilled in the art will appreciate that any number of channels,or layers, to tonal ranges may be separated for individual processingusing the same or differing techniques on individual layers.Additionally, all or only some of the individual layers may be mergedinto a final merged photogrammetric rendering.

Persons skilled in the art will appreciate that end documents or imagesmay be produced in more that one color of ink for the purpose forfurther enhancing the benefits of or use of the end documents to endusers (e.g., highlighting areas of interest or special need orattention). In one example, with respect to FIGS. 9C through 9G, it maybe beneficial to produce the combined image in various colors using thevarious images for each color in order to accentuate certain features(e.g. mortar joints) in combined image. In another example, rectifiedimages of objects that exist in more than one plane may be produced bydisplaying the objects in different planes, such as those notperpendicular to the orthogonal view presented, in different colors.

In the case of the AEC industry, projections from the orthogonal planeof a building that are sloped, slanted, curved or otherwise notorthogonally viewed from the same position may be represented inalternate colors to indicate that in the view presented they are notpresented orthogonally, cannot be measured photgrammetrically, or thearea as shown is different from the true area as viewed or fieldmeasured. Display or printing in multiple line art colors is possible,for example, by either copying the image to a different color channel orby changing the color channel prior to display or by using graphic artsfilms or printing plates with different ink colors than the original.

The advantages of the invention are clearly understood by viewing thedifferences between FIG. 10, prior art FIG. 11, and FIG. 12, while FIG.13 shows processing to retain more or less detail in the image of anobject when preparing a photogrammetric rendering.

FIG. 10 shows detail of the digital photogrammetric images found in FIG.7, shown here in continuous tone grayscale. Window Lintel 1010 showsstone textures 1012 clearly. Also paint drops 1014 are clearly shown.The window trim made of wood serving as rain drip edge 1202 also showscolor reproduction is also possible particularly in the case of lowcolor saturation so that printed documents may be marked and written inthe field by users. Persons skilled in the arts will appreciate that aone-color drawing, or a drawing of any number of colors, may be printedin multiple colors or “full color process” as it is known in the graphicarts industry. For example, particular line segments of a one-color linedrawing may be given different colors such that a full-color documentmay be displayed or printed.

Persons skilled in the art will appreciate that color reproduction,particularly in the case of low color saturation, is also possible sothat printed documents will appear “washed out,” and may be more clearlyand noticeably marked and written on in the field by users.

FIGS. 12A through 12J show the advantages of several embodiments andalternate graphic arts process performed on FIG. 10.

FIG. 12A shows image 1220, which is a grayscale rendering of FIG. 10 in8 levels of gray. The stone texture 1212A and paint drops 1214A areclearly seen. It is also easily determined that 1218A is wood framingaround the window. Lighting rod cable 1216A is clearly seen as somethingother than a mortar joint and most probably a wire or cable. As agrayscale rendering, FIG. 12A may be reproduced by photocopy, laserprinter, printing or other means capable of reproducing continuous toneor near continuous tone images with success.

A process for creating image 1220 of FIG. 12A can be traced in process250 of FIG. 2B. Here, image 500 of FIG. 5 may corresponds to non-metricdigital image 252. Graphic arts process 256 may be to digitally mask theextraneous content so that result is image 600 of FIG. 6. The result isa pre-processed non-metric digital image 258 that is ready for furthergraphic arts processing, not shown in 250, or photogrammetricprocessing. Image 258 is then corrected for perspective and skew 259resulting in a digital photogrammetric image 260 shown as image 700shown in FIG. 7. Perspective correction can either be accomplishedmanually using cameras with nonparallel image display and image receptorfilm planes or using digital photo editing processes of perspectivecorrection.

An example of perspective and skew which are demonstrated in 1460 of 14Fand 1450 of 14E, respectively with the correction resulting as 1410 ofFIG. 14A can be seen in image 600 of FIG. 6 prior to correction andimage 700 of FIG. 7 after correction. Less noticeable is in image 500 ofFIG. 5 is horizon misalignment demonstrated in 1440 of FIG. 14D whichmay also be corrected to 1410 of FIG. 14A as shown in 600 of FIG. 6. Inthis case, image 500 of FIG. 5 is more analogous to 1470 of FIG. 14Gundergoing multiple serial graphic arts processes to result in image 700of FIG. 7.

Person skilled in the art will appreciate that the multiplicity ofeffects shown in FIGS. 14B through 14G, take either individually or as awhole, as may be corrected for serially or in any order such thatcorrected image 1410 of FIG. 14A is achieved. For example, 1470 of FIG.14G may be corrected. It may be desirable, depending upon manual orcomputer-aided processes, to proceed in a particular order, and correctdefects in a particular sequence depending upon both the original imageand the desired end-use image.

Not shown in process 250 is a graphic arts process of extracting,clipping, or cutting a portion of image 700, and then scaling andcropping the extraction so that the resulting image portion correspondsto 1000 in FIG. 10. Image 1000 which, is still a continuous tone imageis then reduced by process 265, in this case multiple processes, namelydiscarding color information, finding edges of contrast change, and thenreducing the image to eight levels of grey, the results beingphotogrammetric rendering 285 shown as image 1220 in FIG. 12A.

FIG. 12B shows a technical drawing style rendering of FIG. 10. Note thatwhile having far less visible detail retained as compared to FIG. 12A,FIG. 12B still shows some of the stone texture 1212B and the paint drops1214B (which were visible in the original digital image FIG. 10). Alsoit certain with a fair level of confidence that 1218 is the wood framingof the window. Image 1240B in FIG. 12B is created by additional graphicarts processing done to image 1220 in FIG. 12A, namely discardinghighlight information and using the edge lines of contrast differentialas guides to draw the lines of the technical drawing. Those familiarwith the art will understand the drawing depicted in image 1240 could becreated by hand drawing, but is more efficiently accomplished usingdigital photo editing techniques.

FIGS. 12C and 12D show alternate further processing of image 1220 fromFIG. 12A with less detail preserved than FIG. 12B. It will be notedhowever that FIG. 12C preserves some of the visual detail of the stonetexture 1212C, that both drawings lose most of the detail associatedwith the paint drops 1214C. Even here though on the visible portions1214C marked on FIG. 12C (and associated structures on FIG. 12D), theanomaly in the stone facade is captured in the drawing. This availabledetail would cause one familiar with the art to inspect the object todetermine what cause the anomaly in the resulting rendering. For a usersuch as a mason, masonry consultant or architect this is a valuablefeature of the invention. Hence even at this level of detail theadditional information retained has value.

FIGS. 13A through 13F shows further processing of FIG. 10 image 1000 toretain more or less detail in the image of an object when preparing aphotogrammetric rendering. FIG. 13A is another result of processingimage 1000 of FIG. 10 to grayscale retaining 8 levels of gray with moredetail available in the shadow.

FIGS. 13B through 13F show the benefits of the combination ofphotogrammetric correction and graphic arts process that are not truerendering process (e.g. ones that are photographic in presentation stylerather than pen and ink drawing styled). Drawings 13B through 13Fprocessing of the image in FIG. 10 to levels that allow reproduction ofthe image on equipment that is capable of producing less and lessdetail. For example FIG. 13A is still an 8 level grayscale image withvery fine detail in the shadows of the original photo still showing e.g.stone texture, and will only reproduce well on an higher end photocopieror laser printer or by traditional printing methods such as lithography.It is not suitable for blueprint or line art reproduction techniques.Image 1320 in FIG. 13A was produced from image 1000 in FIG. 10 with thegraphic arts processes of finding edges of contrast change, discardingcolor information by converting to continuous tone grey scale, adjustingshadow contrast, and reducing to 8 levels of grey. As can be seen theresult clearly show stone surfaces and textures.

FIG. 13B shows FIG. 13A as processed by adjusting the tonal balance toenhance edge definition of the mortar in the masonry facade, reducinggrays to 4 levels and equalizing the remaining levels grey to one level,namely black. The resulting image is effectively line art (e.g. 1 bitcolor) reproducible in one color even as a blueprint. Note the retaineddetail on the stone texture 1312B, the paint drops 1314B, and theclarity that 1318B is wood framing. This FIG. 13B demonstrates the clearhallmark of the invention which is retention of image detail andelimination of undesired detail in a photogrammetric rendering that canbe reproduced as one color line art (e.g., in identical fashion toblueprints used in the AEC industry).

FIG. 13C shows the further processing of FIG. 13A down to 4 levels ofgray. Less detail in available in this image that FIG. 13A and whenfurther processed further to equalize all levels of gray to a singlelevel the result is shown in FIG. 13D.

FIG. 13D, while showing more detail of the stone texture 1312D, has moreextraneous marks 1320D that may actually make the photogrammetricrendering less useful than the one shown in FIG. 13B for certainpurposes. FIG. 13D demonstrates that in the case on one color line artstyle reproduction (e.g. 1 bit color) for blueprints too muchinformation may be retained.

FIGS. 13E and 13F show processing of FIG. 10 to reduce the levels ofgray to 3 and 2 respectively. Two levels of gray shown in FIG. 13F isequivalent to 1 bit color shown in FIG. 13B, but note the absence ofdetail in FIG. 13F as compared to FIG. 13B in the stone texture of thelintel 1312F, window wood frame 1318F, and stone texture 1320F. Thedifferences between FIGS. 13B and 13F show the difference in the orderof graphic arts processes applied and when applied and discussedpreviously in the embodiment 400 shown in FIG. 4.

Persons skilled in the art will appreciate that there are differingneeds for retention or discard of information contained in an imagedepending upon the end users and intended uses of the image, e.g. anarchitect's or artist's stylistic presentation has differentrequirements from a masonry consultant's or mason's requirements. Theflexible iterative system and process described in 400 becomes importantto the invention.

FIG. 13E showing 3 levels of gray (e.g. black, 50% of black, and 0% orabsent) would clearly be reproducible in satisfactory quality on a lowerend printing equipment (e.g., photocopy machine or facsimile machine)without significant loss of detail in the image.

FIG. 14A shows the equivalent of a wire frame projection for anelevation view of an architectural design such as building (e.g.elevation drawing by an architect). An advantage of the invention isthat images, e.g. photographs, with the defects illustrated in FIGS. 14Bthrough 14G either individually, in the case of for example FIGS. 14Bthrough 14F, or multiply, in the case of for example FIG. 14G, can bephotogrammetriclly corrected and rendered.

Perspective and skew defects which are demonstrated in 1460 of FIGS. 14Fand 1450 of FIG. 14E respectively with the correction resulting as 1410of FIG. 14A can be seen in image 600 of FIG. 6 prior to correction andimage 700 of FIG. 7 after correction. Perspective and skew defects canoccur, for example, where a face of an object such as a building wallbeing photographed and an image receptor plane are not parallel to eachother, and in another example where the object of interest is notcentered in the image receptor of a camera.

Less noticeable in image 500 of FIG. 5 is horizon misalignment where,for example, the object of interest and the camera image receptor planeare on differing horizontal horizons as demonstrated in 1440 of FIG.14D. Horizon alignment defects 1440 of FIG. 14D may also be corrected to1410 of FIG. 14A as shown in image 600 of FIG. 6.

Other defects in images may be introduced by the method of image capturesuch as photography when, for example, the camera itself, through theoptics of the camera lens and image receptor system, introducesdistortion into the image captured. Two common examples of camerainduced distortion are barrel lens distortion (e.g., 1420 of FIG. 14B)and pin cushion distortion (e.g., 1430 of FIG. 14C). In both cases,increased distortion may exist further away from the center of thecamera optics, at the outer edges of the image.

Camera induced defects can be partially mitigated, for example, bykeeping the areas of interest of a photograph in the center and awayfrom the edges of the image receptor, as can be seen in 1420 of FIG.14B. It is an advantage of the invention to photogrammetrically correctsuch distortion so that a resulting renderings is analogous to thearchitectural elevation of 1410 of FIG. 14A.

Persons skilled in the art will appreciate that distortions in images tobe photogrammetrically rendered are rarely found signally in originalimages to be processed as demonstrated in FIGS. 14B through 14F. Infact, image 500 is more analogous to 1470 showing a multiplicity ofeffects, and undergoing multiple serial processes to result in image 700which is analogous to 1410. In fact, image 500 after being processed toimage 600 was corrected for pin cushion effect 1430 prior to removal ofperspective 1460 and skew 1450.

Persons familiar with the art will understand that the multiplicity ofeffects shown in FIGS. 14B through 14G, either individually or multiply(present as in the case of 1470 of FIG. 14G) may be corrected seriallyin any order to achieve the corrected image 1410 of FIG. 14A. Dependingupon manual or computer aided processes, it may be desirable to proceeddifferently, and correct defects in a particular sequence depending uponboth the original image and the desired end use image.

The consequent advantage of the various techniques described herein mayinclude, for example: (a) use on non metric images for creation of scalerenderings for the AEC industry, (b) conversion of non metricphotographs to photogrammetric renderings (c) elimination of the needfor specialize metric cameras and equipment, (d) retention ofphotographic or other original image detail in the photogrammetricrendering, (e) reduction of unnecessary detail in photographs to usefullevels for the end user, (f) renderings reproducible by a variety ofprint production methods, (g) reduction of photographs to renderingscapable of being reproduced using blueprint production methods, (h)Economic non contact quick non-labor intensive gathering of fieldinformation, (i) reduction of the need for or quantity of fieldmeasurements, (j) economic reproduction of photogrammetric renderings asblueprints, photo copies or laser prints instead of photographs or largeformat ink jet printing, and/or (k) rapid production of photogrammetricrenderings.

Accordingly, the invention permits the delivery of photogrammetricrenderings and/or documents containing photogrammetric images and otherinformation to end users which said images and/or documents contain moreinformation and are of greater user to end users than images anddocuments currently produced and provided in the AEC industry. In fact,information contained in photographs and images of object of interest tobe processed may be fully or partially retained in a way to that theinvention may achieve, for example: (a) it permits more information tobe displayed visually to end users, (b) it allows end users to moreeasily locate on an object an area of interest in a document image andvice versa, (c) it allows end users with greater certainty to locate onan object areas of interest in a document image and vice versa, (d) itallows for more cost effective, less labor intensive, automated orsemi-automated production of end user documents, and/or (e) it provideseconomic benefits to end users manifested through greater ease of use,greater certainty, and cost of production.

The scope of the invention extends beyond the AEC industry tocartography and other areas where the incorporation of additional visualinformation may enhance ease of use, certainty of the user about objectattributes, user location, the environment or other issues ofimportance.

Inasmuch as the present invention is subject to many variations,modifications and changes in detail, it is intended that all subjectmatter discussed above or shown in the accompanying drawings beinterpreted as illustrative only and not be taken as limiting in anysense.

From the foregoing description, persons skilled in the art willrecognize that this invention provides photogrammetric rendering. Inaddition, persons skilled in the art will appreciate that the variousconfigurations described herein may be combined without departing fromthe present invention. It will also be recognized that the invention maytake many forms other than those disclosed in this specification.Accordingly, it is emphasized that the invention is not limited to thedisclosed methods, systems and apparatuses, but is intended to includevariations to and modifications thereof which are within the spirit ofthe following claims.

What is claimed is:
 1. A method for producing a two-dimensional rendereddrawing of an object, the method comprising: processing a non-metricimage of the object to create an orthoimage; processing original surfacetexture data of the object from the orthoimage to generate as-isprocessed texture data that enhances one or more features of a firstsurface type of the object, wherein the as-is processed texture data isbased entirely on the original surface texture data of the object; andgenerating the two-dimensional rendered drawing of the object thatportrays the as-is processed texture data, wherein the rendered drawingpreserves some detail related to the first surface type of the objectwhile eliminating some detail related to a second surface type of theobject.
 2. The method of claim 1, wherein the processing the non-metricimage of the object to create the orthoimage comprises correcting for acamera lens distortion.
 3. The method of claim 1, wherein the processingthe non-metric image of the object to create the orthoimage comprises atleast one of a perspective correction, a skew correction, and analignment correction.
 4. The method of claim 1, wherein the processingthe original surface texture data of the object from the orthoimage togenerate the as-is processed texture data comprises: determining a tonalthreshold of the orthoimage; and processing the original surface texturedata by isolating the tonal threshold.
 5. The method of claim 1, whereinthe processing the original surface texture data of the object from theorthoimage to generate the as-is processed texture data comprisesfinding edges.
 6. The method of claim 5, wherein the edges comprise atleast one of regions of change in texture, regions of change incontrast, and regions of change in color.
 7. The method of claim 5,wherein the finding edges comprises detecting at least one of regions ofdepth changes, regions of change in material properties, regions ofsurface orientation changes, and regions of change in illumination. 8.The method of claim 1, wherein the as-is processed texture datacomprises texture detail that is reproducible as one-color art whilestill visually indicating the original surface texture data of theobject.
 9. The method of claim 1, wherein the eliminating some detailrelated to the second surface type of the object comprises at least oneof erasure, masking, threshold cutoff, and tonal balance adjustment. 10.The method of claim 1, wherein the processing the original surfacetexture data of the object from the orthoimage and the generating thetwo-dimensional rendered drawing of the object are performedautonomously by a computer.
 11. The method of claim 1, wherein the firstsurface type comprises a first material type and the second surface typecomprises a second material type.
 12. The method of claim 1, wherein thefirst and second surface types comprise identical material types.